Professional course assessments: developmental or regulatory?

Phil Knott, Nottingham Law School

Opening address at Vocational Teachers Forum II, 1 July 2002

The opening address at the second Vocational Teachers Forum explored the relationship and tensions between assessment for learning and assessment for practice through a series of questions designed to promote debate and discussion.

As providers of professional legal education teachers are of course concerned with the relationship between assessment and effective learning. This approach tends to promote innovation and diversity, formative and continuous assessment. However, there may be a tension between this approach and the professions’ legitimate interest in assessment for practice, where issues such as consistency, rigour and ‘fitness to practise’ come more to the fore.

What are vocational assessments for?

I will take it as axiomatic that assessment methods should reflect learning objectives, which in turn should reflect course aims.

The stated aims of the vocational courses are:

• Legal Practice Course (LPC) – to prepare students for practice
• Bar Vocational Course (BVC) – to enable student to become an effective member of the Bar

Looking at it from the students’ perspective, their ultimate objective in attending our courses is to obtain the relevant qualification. They are more interested in the professional award than in our (university) diploma. This recognition should not compromise our aim to produce a high quality learning experience, however, it would be naive to ignore that accreditation is a key feature of our courses in the eyes of the students. For good or ill, this means that assessment is a key driver of their learning experience, as indeed it is across the higher education sector.

Taking into account that the course aims and the student perspective both relate strongly to the world of practice, we need to acknowledge from the outset that the professions have a legitimate interest in the outcomes of our courses. We are all therefore required to operate within a framework that incorporates features such as consistency, objectivity, rigour and individual endeavour. The challenge is to develop diverse and innovative assessments within such a framework.

Can assessments support the learning process?

Initially, our assessment strategy at Nottingham identified assessment as a ‘continuous thread’ running through the course. Assessment would form an integral part of the course, flowing naturally from the learning process. Looking back, this seems from experience to have been a somewhat naive view. For students, assessment is a key driver, and the integration model has in reality often disrupted learning, leading students to become obsessed with the assessment process. This was one of the prime reasons for the Law Society setting up its recent working group on assessment burden, which has led to a reduction in both the number, and continuous nature, of assessments.

Many providers have moved towards placing assessments into clusters, which are thereby separated from teaching. Pragmatically, this recognises that whilst integrating assessments and learning may be the ideal, some level of separation is desirable in order to avoid the potentially disruptive effect that assessments can have on learning.

Can assessments promote reflection?

The idea is frequently mooted that vocational courses should embrace the notion of the ‘reflective practitioner’. Our experience in Nottingham suggests that this view is misplaced. The concept of reflective practice involves acquired knowledge, skills and attitudes being acted on in a practice context in order for reflection, evaluation, planning and improved performance to take place.

Reflection is characteristically assessed through processes involving development portfolios or diaries. These involve personal outcomes, thoughts, feelings, behaviours and explorations of ideas. In practical terms it involves a degree of experimentation, reflection and coaching, which is difficult to replicate in a vocational learning environment. Even more difficult is to assess such learning in a sufficiently consistent and objective manner.

A more realistic objective would be to promote reflective learning. That is learning which through reflection enables students to continue to develop capabilities beyond the immediate (assessment) objective and put it into practice. This is in fact an explicit objective of both the BVC and the LPC.

Reflective learning is a legitimate objective of vocational courses. It does not of course follow that we should incorporate such reflective learning into the core summative assessment process. It would be extremely challenging to develop objective and nationally consistent assessment criteria and provide adequate time and resources to enable students to develop these reflective skills to the level where legitimate assessment could take place.

In a limited way we in fact attempted such a process at Nottingham in our negotiation courses (now of course removed from the LPC). Having conducted a negotiation, students were required to proceed immediately to a closed environment and write up an analysis of their own performance. Thus students’ ability to analyse and reflect upon the strengths and weaknesses of their own performance played a part in the assessment process. In a confrontational skill such as negotiation this was particularly valuable, as students may well have under-performed because of their failure to deal with a difficult or aggressive opponent. An understanding of how this process misfired could rescue a student performance to a level of competence.

By way of summary, we should certainly encourage and develop reflection wherever possible; at least by ensuring that assessment overload does not inhibit and undermine the motivation and opportunity for reflective learning. Whilst limited assessment opportunities might present themselves, we should in my view resist the temptation to make reflective learning a core part of the assessment process.

What assessment methods are available?

Turning to a more practical issue, providers do not always explore the full range of available instruments when drafting assessments. Remembering that we must ensure that we match learning objectives with assessment methods, a quick personal brainstorm suggests that there is a very substantial range of options available:

• advanced materials
• closed book
• computer-based
• extended time frame
• knowledge examination
• MCQs
• narrative assessment
• open book
• peer assessment
• reflective log
• report
• self assessment
• short answer questions
• simulation
• take away
• task based
• time restricted
• transactional examination

Are particular assessment methods intrinsically more suited to developing different cognitive skills? Traditionalists might argue that methods such as multiple choice and short answer questions can only address knowledge, comprehension and application. However, there is increasing support for using such methods to assess higher level cognitive skills such as analysis, synthesis and evaluation. I think we need a real debate on the extent to which ‘efficient’ (a euphemism for ‘easy to mark’) assessment methods can in fact reach all parts of Bloom’s taxonomy.

Some specifics on BVC and LPC assessment methods

What then are the essential similarities and differences between the BVC and LPC? As Amanda Fancourt has produced a thorough comparison of the two assessment regimes, I will confine myself to one or two specific observations:

• from September 2002 both courses will have 14 assessment points
• the BVC identifies its substantive litigation subjects as ‘knowledge areas’. Assessments are normally multiple choice or short answer questions designed primarily to elicit knowledge, comprehension and application. LPC substantive subjects are far more disparate in content. Further, they are typically assessed in a more transactional context involving case studies, which at first sight give greater scope for deep learning, and for assessing the higher cognitive skills of analysis, synthesis and evaluation. However, as I have argued above, both courses would benefit from considering a wider range of assessment instruments, and evaluating how most effectively to assess the various levels of student learning.
• BVC examinations are closed book; LPC assessments are normally (but not exclusively) open book
• the BVC skills assessments are ascribed a specific mark, whereas LPC skills are merely competent/not competent

Can assessment incorporate an holistic element?

One final issue before I conclude is whether the assessment process can develop any form of holistic outcome. In addressing this, I am greatly taken by the notion of an ‘exit assessment’, as practised at the Institute of Professional Legal Studies in Belfast, which can develop students’ overall understanding and skills obtained throughout the course in a neutral context. This would of course be an ambitious project, but is worthy of careful evaluation. Arguably it should be a national assessment. Given the diversity of the legal profession in England and Wales, I would argue that the appropriate place for this would be at the relatively common LPC stage, rather than at the end of the training contract (as in Scotland).

What are the components of a good assessment?

By way of conclusion, here is my checklist for a ‘good’ assessment. It would not of course be possible (or even desirable) to incorporate all such features into a single assessment. Rather, it might be an aspiration for an overall diet of assessments:

• fairness
• consistency
• objectivity
• a range of learning outcomes
• efficiency
• flexibility
• innovation
• holism
• reflection

Last Modified: 4 June 2010